Papers by Bill Yuchen Lin

Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality (2022.emnlp-main)

Challenge:	a small learning gap exists between large and small language models . long CoT data and large model responses are not beneficial for small models - a problem that may be due to the small student model's ability to handle distribution shifts.
Approach:	They propose a mix distillation strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models.
Outcome:	The proposed strategy outperforms training on large and small models on short CoT and small model CoT.

Copied to clipboard

Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, Xiang Ren

Dialogue Systems & Conversational Agents

Challenge:	Currently, human communication models fail to explicitly model common ground (CG) . less than half of the responses in current data is rated as high quality .
Approach:	They propose a dataset that annotates dialogues with explicit CG and solicits 9k diverse responses each following one common ground.
Outcome:	The proposed dataset annotates dialogues with explicit CG and solicits 9k diverse responses each following one common ground.

Probing Commonsense Explanation in Dialogue Response Generation (2021.findings-emnlp)

Copied to clipboard

Pei Zhou, Pegah Jandaghi, Hyundong Cho, Bill Yuchen Lin, Jay Pujara, Xiang Ren

Dialogue Systems & Conversational Agents

Challenge:	Currently, response generation (RG) models do not understand human communication intents.
Approach:	They propose to examine commonsense reasoning implicitly to determine whether RG models produce coherent responses in conversations.
Outcome:	The proposed probing settings show that RG models fail to capture the logical relations between commonsense explanations and responses and fine-tuning on in-domain data do not lead to understanding of CSR for RG.

RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models (2021.emnlp-main)

Copied to clipboard

Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, Xiang Ren

Robustness in NLP

Challenge:	Recent named entity recognition models have great performance on many conventional benchmarks, but it is not reliable in realistic applications.
Approach:	They propose a method to create natural adversarial examples using Wikidata and pre-trained language models.
Outcome:	The proposed method produces natural adversarial examples with a shifted distribution from training data.

On the Robustness of Reading Comprehension Models to Entity Renaming (2022.naacl-main)

Copied to clipboard

Jun Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, Xiang Ren

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020.findings-emnlp)

Challenge:	SpanBERT model is more robust than RoBERTa, despite having similar accuracy on unperturbed test data.
Approach:	They propose a pipeline to replace entity names with names from a variety of sources.
Outcome:	The proposed model performs worse when entities are renamed, the authors show . SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa .

Copied to clipboard

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, Xiang Ren

KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (D19-1)

Challenge:	Recent studies show that pre-trained language models perform well on commonsense-reasoning benchmark datasets, but building machines with commonsence to compose plausible sentences remains challenging.
Approach:	They propose a constrained text generation task for generative commonsense reasoning that generates a coherent sentence using common concepts.
Outcome:	The proposed task generates a coherent sentence describing an everyday scenario using common concepts over 35k concept-sets.

Copied to clipboard

Bill Yuchen Lin, Xinyue Chen, Jamin Chen, Xiang Ren

Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024.emnlp-main)

Challenge:	empowering machines with the ability to perform commonsense reasoning has been seen as the bottleneck of artificial general intelligence .
Approach:	They propose a textual inference framework that uses external commonsense knowledge graphs to answer commonsensical questions.
Outcome:	The proposed framework is based on graph convolutional networks and LSTMs with a hierarchical path-based attention mechanism.

Copied to clipboard

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo

Selective “Selective Prediction”: Reducing Unnecessary Abstention in Vision-Language Reasoning (2024.findings-acl)

Challenge:	Existing open-source evaluation paradigms lack flexibility and performance . language model-based evaluation is cheap and scalable, but it is difficult to evaluate .
Approach:	They propose a language model-based evaluation paradigm that uses a scalar indicator of quality to assess LM outputs.
Outcome:	The proposed language model-based evaluation model is more powerful than its predecessor.

Copied to clipboard

Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Chandu

Multimodality

Challenge:	ReCoVERR reduces the over-abstention of a vision-language system with low tolerance for inaccurate predictions without increasing the error rate of the system’s predictions.
Approach:	They propose an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system’s predictions.
Outcome:	ReCoVERR reduces the over-abstention of a vision-language system without increasing the error rate of the system’s predictions.

RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge (2021.findings-acl)

Copied to clipboard

Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, Xiang Ren

LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation (2020.acl-demos)

Challenge:	a riddle-style commonsense questions require complex commonsensense reasoning and figurative language skills . there is currently no dataset aimed at testing these abilities . authors propose a new multiple-choice question answering task .
Approach:	They propose a new multiple-choice question answering task that uses a large dataset for riddlestyle commonsense questions.
Outcome:	The proposed task comes with the first large dataset for answering riddlestyle commonsense questions.

Copied to clipboard

Dong-Ho Lee, Rahul Khanna, Bill Yuchen Lin, Seyeon Lee, Qinyuan Ye, Elizabeth Boschee, Leonardo Neves, Xiang Ren

Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning (2021.findings-emnlp)

Challenge:	Existing frameworks for sequence labeling and classification require massive human effort and labeling data is limited.
Approach:	They propose a web-based, Label-Efficient AnnotatioN framework that allows an annotator to provide the needed labels for a task and can capture explanations for each labeling decision.
Outcome:	The proposed framework surpasses baseline F1 scores by 5-10 percentage points while using 2X times fewer labeled instances.

Copied to clipboard

Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, Xiang Ren

Differentiable Open-Ended Commonsense Reasoning (2021.naacl-main)

Challenge:	Existing models that pursue rapid generalization to new tasks are mostly trained in a single shot on fixed datasets, unable to dynamically expand their knowledge.
Approach:	They propose a new learning setup that assumes a model learns from a sequence of diverse NLP tasks arriving sequentially, accumulating knowledge for improved generalization to new tasks.
Outcome:	The proposed learning setup improves generalization ability while retaining performance on the tasks learned earlier.

Copied to clipboard

Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William Cohen

CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)

Challenge:	Existing commonsense reasoning models work by scoring a question-candidate pair, but new approaches are needed to answer multiple-choice questions.
Approach:	They propose to use a corpus of commonsense facts to answer a commonsensical question without any pre-defined choices as a resource.
Outcome:	The proposed model outperforms baseline methods by a large margin in the open-ended commonsense reasoning task.

Copied to clipboard

Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi

Robustness in NLP

Challenge:	CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru.
Approach:	They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru.
Outcome:	The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East.

Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering (2020.emnlp-main)

Copied to clipboard

Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, Xiang Ren

Question Answering

Challenge:	Existing work on augmenting question answering models with external knowledge (e.g., knowledge graphs) lacks transparency into the model’s prediction rationale.
Approach:	They propose a knowledge-aware approach that equips pre-trained language models with a multi-hop relational reasoning module that performs multi-relational reasoning over subgraphs extracted from external knowledge graphs.
Outcome:	The proposed model performs multi-hop, multi-relational reasoning over subgraphs extracted from external knowledge graphs.

Complex Reasoning in Natural Language (2023.acl-tutorials)

Copied to clipboard

Wenting Zhao, Mor Geva, Bill Yuchen Lin, Michihiro Yasunaga, Aman Madaan, Tao Yu

Mining Cross-Cultural Differences and Similarities in Social Media (P18-1)

Challenge:	Recent research shows that pretrained language models are often brittle for complex reasoning tasks.
Approach:	They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks .
Outcome:	This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness .

Copied to clipboard

Bill Yuchen Lin, Frank F. Xu, Kenny Zhu, Seung-won Hwang

Multilinguality

Challenge:	a new paper examines the problem of computing cross-cultural differences and similarities in natural language understanding . cross-culture differences are important for cross-lingual research, especially in social media .
Approach:	They propose a framework for computing cross-cultural differences and similarities from social media . they propose to use a social media platform to find similar terms for slang across languages .
Outcome:	The proposed framework outperforms baseline methods on two novel tasks.

TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition (2020.acl-main)

Copied to clipboard

Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, Xiang Ren

Agent Lumos: Unified and Modular Training for Open-Source Language Agents (2024.acl-long)

Challenge:	Named entity recognition (NER) is a fundamental information extraction task that focuses on extracting entities from a given text and classifying them using pre-defined categories.
Approach:	They propose to use “entity triggers” to facilitate label-efficient learning of NER models.
Outcome:	The proposed model is significantly more cost-effective than the traditional neural NER frameworks.

Copied to clipboard

Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, Bill Yuchen Lin

Challenge:	Lumos is a framework for training open-source agents on complex interactive tasks.
Approach:	They propose a framework for training open-source LLM-based agents called Lumos . Lumos features a learnable, unified and modular architecture with a planning module that learns high-level subgoal generation and a grounding module trained to translate these into the actions using various tools in the execution module.
Outcome:	The framework outperforms open-source agents on QA and web tasks.

Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models (2020.emnlp-main)

Copied to clipboard

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, Xiang Ren

VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (2024.emnlp-main)

Challenge:	Recent studies show that pre-trained language models possess certain commonsense and factual knowledge.
Approach:	They propose to use pre-trained language models to predict masked words . they introduce a probing task with 13.6k m-word-prediction probes .
Outcome:	The proposed model performs poorly on the diagnostic dataset prior to any fine-tuning and fine-testing with distant supervision.

Copied to clipboard

Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Rongqi Fan, Zhiheng Lyu, Bill Yuchen Lin, Wenhu Chen

Visual Data in NLP

Challenge:	Existing video metrics are lagging behind in providing reliable scores over generated videos due to lack of large-scale human-annotated dataset.
Approach:	They propose to use VideoFeedback to train a human-annotated multi-aspect score over 37.6K synthesized videos from 11 existing video generative models.
Outcome:	The proposed model outperforms the prior best metrics by 50 points in the test.

Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning (2021.acl-long)

Copied to clipboard

Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, Xiang Ren

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021.emnlp-main)

Challenge:	Using multilingual language models, commonsense reasoning research has been limited to English.
Approach:	They propose a Mickey Probe task to evaluate commonsense across languages . they propose X-CSQA and XCODAH datasets to be translated to 14 languages based on the Mickey corpus .
Outcome:	The proposed method significantly improves sentence representations beyond English.

Copied to clipboard

Qinyuan Ye, Bill Yuchen Lin, Xiang Ren

AutoTriggER: Label-Efficient and Robust Named Entity Recognition with Auxiliary Trigger Extraction (2023.eacl-main)

Challenge:	We study whether and how cross-task generalization ability can be acquired . we use CrossFit to standardize seen/unseen task partitions and evaluation protocols .
Approach:	They propose a problem setup for studying cross-task generalization ability which standardizes seen/unseen task partitions and data access during different learning stages.
Outcome:	The proposed model can be used to build few-shot learners across diverse tasks.

Copied to clipboard

Dong-Ho Lee, Ravi Kiran Selvam, Sheikh Muhammad Sarwar, Bill Yuchen Lin, Fred Morstatter, Jay Pujara, Elizabeth Boschee, James Allan, Xiang Ren

FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks (2022.findings-naacl)

Challenge:	Named entity recognition models have shown impressive results in overcoming label scarcity and generalizing to unseen entities by leveraging distant supervision and auxiliary information such as explanations.
Approach:	They propose a framework that automatically generates and leverages “entity triggers” which are human-readable cues in the text that help guide the model to make better decisions.
Outcome:	The proposed framework outperforms the RoBERTa-CRF baseline by nearly 0.5 F1 points on three well-studied datasets.

Copied to clipboard

Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, Salman Avestimehr

LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023.acl-long)

Challenge:	Increasing concerns and regulations about data privacy necessitate the study of privacy-preserving, decentralized learning methods for natural language processing tasks.
Approach:	They propose a framework for evaluating federated learning methods on four different tasks . they propose federation between Transformer-based language models and FL methods .
Outcome:	The proposed framework compares FL methods on four different tasks under non-IID partitioning strategies.

Copied to clipboard

Dongfu Jiang, Xiang Ren, Bill Yuchen Lin

Knowledge-Augmented Methods for Natural Language Processing (2022.acl-tutorials)

Challenge:	a recent study shows that open-source large language models (LLMs) exhibit diverse strengths and weaknesses due to variations in their architectures and training data.
Approach:	They propose a framework that leverages the diverse strengths of open-source large language models.
Outcome:	The proposed framework outperforms individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

Copied to clipboard

Chenguang Zhu, Yichong Xu, Xiang Ren, Bill Yuchen Lin, Meng Jiang, Wenhao Yu

Stronger Models are Not Always Stronger Teachers for Instruction Tuning (2025.naacl-long)

Challenge:	Knowledge in natural language processing (NLP) is a rising trend especially after the advent of large scale pre-trained models.
Approach:	This tutorial introduces the key steps in integrating knowledge into natural language processing (NLP) it introduces knowledge grounding from text, knowledge representation and fusing.
Outcome:	This tutorial introduces the key steps in integrating knowledge into natural language processing including knowledge grounding from text, knowledge representation and fusing.

Copied to clipboard

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Radha Poovendran

SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)

Challenge:	Existing methods to optimize instruction-following capabilities of large language models (LLMs) assume that larger or stronger models are stronger teachers and therefore adopt smaller models as response generators.
Approach:	They propose to use large-scale instruction datasets to tune large language models to align with specific tasks and user intents.
Outcome:	The proposed metric outperforms most baselines in identifying the effectiveness of response generators.

Copied to clipboard

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran

ExtRA: Extracting Prominent Review Aspects from Customer Feedback (D18-1)

Challenge:	Despite advances in large language models, they face substantial challenges in terms of safety.
Approach:	They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks.
Outcome:	The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs.

Copied to clipboard

Zhiyi Luo, Shanshan Huang, Frank F. Xu, Bill Yuchen Lin, Hanyuan Shi, Kenny Zhu

Information Extraction & Text Mining

Challenge:	Existing methods for analyzing and summarizing customer reviews are based on a number of prominent review aspects.
Approach:	They propose a framework for extracting the most prominent aspects of a given product type from textual reviews.
Outcome:	The proposed framework extracts K most prominent aspect terms which do not overlap semantically without supervision.

Temporal Sampling for Forgotten Reasoning in LLMs (2026.acl-long)

Copied to clipboard

Yuetai Li, Zhangchen Xu, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Xiang Yue, Radha Poovendran

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)

Challenge:	a new metric measures the percentage of questions that were answered incorrectly during fine-tuning .
Approach:	They propose a decoding strategy that draws outputs from multiple checkpoints along the training trajectory.
Outcome:	The proposed method improves reasoning performance and consistency across benchmarks.

Copied to clipboard

Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo

SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities (2025.findings-acl)

Challenge:	a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment.
Approach:	They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation .
Outcome:	The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks.

Copied to clipboard

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, Radha Poovendran

The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism (2025.naacl-long)

Challenge:	Emerging large reasoning models (LRMs) leverage long chain-of-thought (CoT) reasoning to enhance their reasoning capabilities.
Approach:	They conduct a systematic study of LRM safety using human annotations to assess their safety.
Outcome:	The proposed safety measures are compared to state-of-the-art models on strong and wildjailbreak datasets.

Copied to clipboard

Yifan Song, Guoyin Wang, Sujian Li, Bill Yuchen Lin

Challenge:	Current evaluations of large language models (LLMs) focus on a single output per example, which limits our understanding of LLM performance variability in real-world applications.
Approach:	They explore the performance differences between greedy decoding and sampling and identify benchmarks’ consistency regarding non-determinism and examine unique model behaviors.
Outcome:	The proposed model outperforms sampling methods and greedy decoding outperformed other models.

On Continual Model Refinement in Out-of-Distribution Data Streams (2022.acl-long)

Copied to clipboard

Bill Yuchen Lin, Sida Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, Scott Yih

Challenge:	Existing continual learning (CL) problems cannot cover real-world scenarios such as out-of-distribution errors.
Approach:	They propose a continual model refinement problem formulation to solve this problem . they extend several existing continual learning approaches to the CMR problem based on a general sampling algorithm .
Outcome:	The proposed model refinement solution improves on existing models and their performance metrics.

AlpacaTag: An Active Learning-based Crowd Annotation Framework for Sequence Tagging (P19-3)

Copied to clipboard

Bill Yuchen Lin, Dong-Ho Lee, Frank F. Xu, Ouyu Lan, Xiang Ren

Tagging

Challenge:	Existing sequence annotation tools focus on improving user interfaces and user interface.
Approach:	They propose an open-source web-based data annotation framework for sequence tagging tasks . the framework is based on active learning and automatic crowd consolidation .
Outcome:	The proposed framework is a comprehensive solution for sequence labeling tasks . it can be deployed in downstream systems while new annotations are being made .

RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)

Copied to clipboard

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi

Neural Adaptation Layers for Cross-domain Named Entity Recognition (D18-1)

Challenge:	Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models.
Approach:	They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries .
Outcome:	The proposed dataset compares RMs with other models on a set of questions.

Copied to clipboard

Bill Yuchen Lin, Wei Lu

Information Extraction & Text Mining

Challenge:	Named entity recognition is a type of information extraction task whereby features can be designed based on domain-specific knowledge.
Approach:	They propose to use existing neural architectures to adapt to new domains without retraining . they propose to add adaptation layers to existing neural models to minimize re-training based on source data.
Outcome:	The proposed approach significantly outperforms state-of-the-art methods on social media domains.

RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms (2021.emnlp-main)

Copied to clipboard

Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, Xiang Ren

Robustness in NLP

Challenge:	Pre-trained language models have impressive performance on commonsense inference benchmarks, but their ability to make robust inferences is debated.
Approach:	They propose a challenge that evaluates robust commonsense inference despite textual perturbations using commonsensical knowledge bases and probe PTLMs across two different evaluation settings.
Outcome:	The proposed procedure evaluates robust commonsense inference despite textual perturbations using commonsensense knowledge bases and probe PTLMs across two evaluation settings.

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (2025.naacl-long)

Copied to clipboard

Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Maria Antoniak, Bill Yuchen Lin, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin Choi

SimulBench: Evaluating Language Models with Creative Simulation Tasks (2025.findings-naacl)

Challenge:	Lack of transparency in training data is limiting external oversight and inspection of LLMs for issues such as copyright infringement and data contamination.
Approach:	They propose a method to identify training data known to proprietary LLMs without requiring access to model weights or token probabilities by using information-guided probes.
Outcome:	The proposed method can identify training data known to proprietary LLMs without access to model weights or token probabilities.

Copied to clipboard

Qi Jia, Xiang Yue, Tuney Zheng, Jie Huang, Bill Yuchen Lin

Automatic Extraction of Commonsense LocatedNear Knowledge (P18-2)

Challenge:	Existing benchmarks for large language models do not fully evaluate their potential for broad implementation.
Approach:	They propose to use a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks.
Outcome:	The proposed framework outperforms LLaMA-3-70b-Chat on 18.55% more cases.

Copied to clipboard

Frank F. Xu, Bill Yuchen Lin, Kenny Zhu

Information Extraction & Text Mining

Challenge:	LocatedNear relation is a kind of commonsense knowledge describing two physical objects that are typically found near each other in real life.
Approach:	They propose to automatically extract LocatedNear relation from corpus by a sentence-level relation classifier and aggregating scores of entity pairs from a large corpus.
Outcome:	The proposed method can be used to extract the commonsense LOCATEDNEAR relation from a large corpus.

Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling (2020.acl-main)

Copied to clipboard

Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, Xiang Ren